22 Sep 2021

Why

investigating badges and trust

  • detrimental effect of replication crisis on perceived trustworthiness (Anvari and Lakens 2018; Wingen, Berkessel, and Englich 2020)


  • community: increase transparency and reproducibility (Lindsay 2015; Vazire 2018)


preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Why

investigating badges and trust

  • detrimental effect of replication crisis on perceived trustworthiness (Anvari and Lakens 2018; Wingen, Berkessel, and Englich 2020)


  • community: increase transparency and reproducibility (Lindsay 2015; Vazire 2018)


  • journals: badges to signal adherence to open science practices (OSP) (list see https://www.cos.io/initiatives/badges)


  • effectiveness to foster the implementation of OSP (Kidwell et al. 2016)



preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Why

investigating badges and trust

  • detrimental effect of replication crisis on perceived trustworthiness (Anvari and Lakens 2018; Wingen, Berkessel, and Englich 2020)


  • community: increase transparency and reproducibility (Lindsay 2015; Vazire 2018)


  • journals: badges to signal adherence to open science practices (OSP) (list see https://www.cos.io/initiatives/badges)


  • effectiveness to foster the implementation of OSP (Kidwell et al. 2016)


  not much is known on the effects of badges at an individual level such as trust  

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Method

Method

Samples

  • Study 1: Undergraduates (pre-service teachers)
    • \(N\) = 270
    • age: \(M\) = 22.9 (3.0); semester: \(M\) = 5.9 (3.7)

  • Study 2: Social Scientists
    • \(N\) = 250
    • position: \(Mo\) = graduate research assistant or postgraduate researcher;
      age: \(Mo\) = younger than 35 years

  • Study 3: General Public (UK)
    • \(N\) = 257
    • cross quota (based on census) of sex, age, qualification

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Method

Measures & Analyses


Measures:
Integrity subscale of epistemic trust (Hendriks, Kienhues, and Bromme 2015)
\(\omega\) = .83-.92



Preregistered:

approximate adjusted fractional Bayes factors (Gu, Mulder, and Hoijtink 2018; Hoijtink et al. 2019)

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Results



preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Results



BF against \(H_0 = 3.5*10^7\), \(H_2 = 4.5*10^1\), \(\overline{H_1} = 4.8*10^3\), \(H_u = 5.5\)
\(d_{GB/CC} = .32\), \(d_{CC/CB} = .29\), \(d_{GB/CB} = .57\)

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Results



BF against \(H_0 = 1.6*10^{11}\), \(H_2 = 7.5\), \(\overline{H_1} = 8.0*10^2\), \(H_u = 5.4\)
\(d_{GB/CC} = .55\), \(d_{CC/CB} = .25\), \(d_{GB/CB} = .77\)

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Results



BF against \(H_0 = 3.2\), \(H_1 = 5.8\), \(\overline{H_2} = 18.5\), \(H_u = 18.5\)
\(d_{GB/CC} = .21\), \(d_{CC/CB} = -.02\), \(d_{GB/CB} = .20\)

preprint bit.ly/OS-Badges     |     slides bit.ly/meta-slides     |     artzyatfailing2

Thank you

tools used

Anvari, Farid, and Daniël Lakens. 2018. “The Replicability Crisis and Public Trust in Psychological Science.” Comprehensive Results in Social Psychology 3 (3): 266–86. https://doi.org/10.1080/23743603.2019.1684822.

Gu, Xin, Joris Mulder, and Herbert Hoijtink. 2018. “Approximated Adjusted Fractional Bayes Factors: A General Method for Testing Informative Hypotheses.” British Journal of Mathematical and Statistical Psychology 71 (2): 229–61. https://doi.org/10.1111/bmsp.12110.

Hendriks, Friederike, Dorothe Kienhues, and Rainer Bromme. 2015. “Measuring Laypeople’s Trust in Experts in a Digital Age: The Muenster Epistemic Trustworthiness Inventory (METI).” Edited by Jelte M. Wicherts. PLOS ONE 10 (10). https://doi.org/10.1371/journal.pone.0139309.

Hoijtink, Herbert, Joris Mulder, Caspar van Lissa, and Xin Gu. 2019. “A Tutorial on Testing Hypotheses Using the Bayes Factor.” Psychological Methods, February. https://doi.org/10.1037/met0000201.

Kidwell, Mallory C., Ljiljana B. Lazarević, Erica Baranski, Tom E. Hardwicke, Sarah Piechowski, Lina-Sophia Falkenberg, Curtis Kennett, et al. 2016. “Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency.” PLoS Biology 14 (5): 1002456. https://doi.org/10.1371/journal.pbio.1002456.

Lindsay, D. Stephen. 2015. “Replication in Psychological Science.” Psychological Science 26 (12): 1827–32. https://doi.org/10.1177/0956797615616374.

Vazire, Simine. 2018. “Implications of the Credibility Revolution for Productivity, Creativity, and Progress.” Perspectives on Psychological Science 13 (4): 411–17. https://doi.org/10.1177/1745691617751884.

Wingen, Tobias, Jana B. Berkessel, and Birte Englich. 2020. “No Replication, No Trust? How Low Replicability Influences Trust in Psychology.” Social Psychological and Personality Science 11 (4): 454–63. https://doi.org/10.1177/1948550619877412.

Additional slides

Treatment Check

Study 1

Treatment Check

Study 2

Treatment Check

Study 3